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SUMMARY 


This  project  carried  out  an  interdisciplinary  research  program  among  scientists  and  stu¬ 
dents  of  the  Boston  Consortium  for  Behavioral  and  Neural  Studies,  which  includes  investi¬ 
gators  from  seven  Boston-area  institutions.  Projects  concerning  the  cognitive,  perceptual, 
and  neural  bases  of  skilled  performance  included  both  experimental  and  theoretical  studies 
of  vision,  speech,  adaptive  pattern  recognition,  attentive  cognitive  information  processing, 
reinforcement  learning  and  prediction,  and  adaptive  sensory-motor  control  and  planning. 
These  studies  paid  particular  attention  to  those  properties  of  biological  intelligence  that  can 
autonomously  adapt  in  real-time  to  unexpected  events.  Major  progress  was  made  towards 
discovering  and  characterizing  neural  network  architectures  in  all  of  the  project  areas. 

The  Boston  Consortium  engaged  in  a  number  of  collaborative  ventures  that  crossed  both 
disciplinary  and  institutional  lines.  These  interactions  were  facilitated  by  providing  offices 
for  Consortium  members  at  the  Center  for  Adaptive  Systems  of  Boston  L  niyersity.  Research 
appointments  and  parking  privileges  at  Boston  University  were  also  provided  to  facilitate 
interactions  of  faculty  from  several  universities. 

For  example,  Professor  James  Todd  of  Brandeis  L  niversity  developed  part  of  his  psy¬ 
chophysics  lab  at  Boston  University  and  hired  a  graduate  student  in  the  Boston  University 
graduate  program  in  Cognitive  and  Neural  Systems  as  an  RA.  Todd  also  joined  Stephen 
Grossberg,  Ennio  Mingolla.  and  Dejan  Todorovic  of  Boston  University  in  a  research  seminar 
on  motion  perception  that  led  to  a  great  deal  of  new  research.  Professor  Paola  Bressan  of 
Padua  University  visited  Boston  University  for  a  year,  and  could  thus  carry  out  an  experi¬ 
mental  collaboration  with  James  Todd  as  well  as  with  Michael  Rudd,  who  is  a  Boston  Uni¬ 
versity  postdoctoral  fellow.  Bressan  and  Rudd  experimentally  tested,  with  positive  results, 
theoretical  predictions  of  the  neural  network  motion  perception  model  of  Stephen  Grossberg, 
Ennio  Mingolla.  and  Michael  Rudd.  James  Todd  and  Robin  Akerstrom  of  Brandeis  also  suc¬ 
cessfully  tested  predictions  about  3-D  shape-from-texture  from  the  neural  network  model  of 
visual  form  perception  that  was  developed  by  Grossberg  and  Mingolla. 

Professor  Adam  Reeves  of  Northeastern  University  joined  Michael  Rudd  and  Stephen 
Grossberg  in  a  theoretical  analysis  of  temporal  properties  of  peripheral  vision.  nota.bly  tran¬ 
sient  tritanopia,  in  which  Reeves  is  a  leading  experimentalist.  Reeves  also  experimentally 
tested  the  prediction  of  Gail  Carpenter  and  Stephen  Grossberg  that  an  attentional  gam 
control  channel  helps  to  control  attentional  priming  and  visual  pattern  recognition.  Reeves, 
with  Larry  Arend  of  the  Eye  Research  Institute,  tested  properties  of  the  neural  network 
model  of  brightness  perception  under  variable  illumination  that  wa  -  u>_  .  eloped  by  Grossberg 
and  Dejan  Todorovic,  a  Boston  University  postdoctoral  fellow. 

Professor  John  Daugman  of  Harvard  University  began  neurobioi^gical  experiments  to 
test  properties  of  the  Grossberg-Mingolla  visual  form  perception  theory,  also  called  the 
Boundary  Contour  System,  or  BCS.  Daugman  also  developed  an  effective  texture  segmen¬ 
tation  algorithm  for  image  processing  that  uses  some  BCS  mechanisms,  as  well  as  a  neural 
network  model  for  learned  orthogonalization  of  non-orthogonal  complete  representations  for 
image  processing,  which  is  closely  related  to  the  scheme  for  adaptive  coordinate  change  that 
was  developed  by  Grossberg  and  Michael  Kuperstein,  then  a  Boston  University  postdoctoral 
fellow,  for  use  in  adaptive  sensory-motor  control.  Michael  Kuperstein  has  since  founded 
a  company  called  Neurogen  to  build  adaptive  robots  whose  controllers  are  based  upon  his 
research  with  Grossberg. 

A  wide  variety  of  such  productive  interactions  has  developed  during  the  U  R1  grant  pe¬ 
riod,  and  have  contributed  to  a  lively  URI-based  intellectual  community  in  Boston.  The 
cooperative  organization  of  this  community  has  made  it  possible  for  its  members  to  initiate 
and  develop  a  large  number  of  research  ties  with  universities,  as  well  as  government  and  in¬ 
dustrial  laboratories  throughout  the  United  States  and  abroad.  For  example,  neural  network 
models  developed  at  the  Center  for  Adaptive  Systems  are  now  being  used  in  cutting-edge 


applications  to  vision,  speech  and  radar,  pattern  recognition,  and  robotics  around  the  world. 

The  dynamism  of  this  community  also  enabled  it  to  play  a  key  role  in  helping  to  develop 
an  infrastructure  for  international  cooperation  in  neural  network  research  and  education. 
These  contibutions  include  founding  the  International  Neural  Network  Society,  founding  the 
journal  Neural  Networks ,  and  helping  to  organize  the  IEEE  First  International  Conference 
on  Neural  Networks. 

More  detailed  summaries  of  URI  activities  are  provided  below. 
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PART  I 


Boston  University  Summary: 

Research  at  the 
Center  for  Adaptive  Systems 
and  the 

Graduate  Program  in  Cognitive  and  Neural  Systems 
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1.  PUBLICATIONS  PARTIALLY  SUPPORTED  BY 
THE  AIR  FORCE  OFFICE  OF  SCIENTIFIC  RESEARCH 

CONTRACT  AFOSR  F49620-87-C-0018 

NOVEMBER  1,  1986-DECEMBER  31,  1989 


BOOKS 

1.  Grossberg.  S.  (Ed.)  (1987).  The  adaptive  brain,  I:  Cognition,  learning,  reinforce¬ 
ment,  and  rhythm.  Amsterdam:  Elsevier/North-Holland.  (x+) 

2.  Grossberg,  S.  (Ed.)  (1987).  The  adaptive  brain,  II:  Vision,  speech,  language,  and 
motor  control.  Amsterdam:  Elsevier/Xorth-Holland.  (*+) 

3.  Grossberg.  S.  (Ed.)  (1988).  Neural  networks  and  natural  intelligence.  Cambridge. 
MA:  MIT  Press.  (*+) 

4.  Grossberg.  S.  and  Kuperstein.  M.  (1989).  Neural  dynamics  of  adaptive  sensory-motor 
control:  Expanded  edition.  Elmsford.  NY:  Pergamon  Press.  (x-f) 


ARTICLES 

1.  Banquet,  J.-P.  and  Grossberg,  S.  (1987).  Probing  cognitive  processes  through  the  struc¬ 
ture  of  event-related  potentials  during  learning:  An  experimental  and  theoretical  analysis. 
Applied  Optics.  26.  4931-4946.  (  +  ) 

2.  Bullock.  D..  Carpenter.  G.A..  and  Grossberg.  S.  (19S9).  Self-organizing  neural  network 
architectures  for  adaptive  pattern  recognition  and  robotics..  In  \  .  Milutinovic  and  P.  An- 
tognetti  (Eds.).  Neural  networks:  Concepts,  applications,  and  implementations. 
Englewood  Cliffs,  NJ:  Prentice  Hall.  (x+) 

3.  Bullock,  D.  and  Grossberg.  S.  (19S7).  A  neural  network  architecture  for  automatic  tra- 
formation  and  coordination  of  multiple  effectors  during  variable-speed  arm  movemei; 

M.  Caudill  and  C.  Butler  (Eds.).  Proceedings  of  the  IEEE  international  conference 
on  neural  networks,  IV,  539-566.  (+) 

4.  Bullock.  D.  and  Grossberg.  S.  (1988).  Neural  dynamics  of  planned  arm  movements:  Emer¬ 
gent  invariants  and  speed-accuracy  properties  during  trajectory  formation.  Psychological 
Review ,  95,  49-90.  (  +  ) 

5.  Bullock,  D.  and  Grossberg,  S.  (1988).  Self-organizing  neural  architectures  for  eye  move¬ 
ments,  arm  movements,  and  eye-arm  coordination.  In  H.  Haken  (Ed.).  Neural  and  syner¬ 
getic  computers.  New  York:  Springer- Yerlag.  197-228.  (  +  ) 

6.  Bullock,  D.  and  Grossberg.  S.  (19S9).  VUE  and  FLETE:  Neural  modules  for  trajectory  for¬ 
mation  and  postural  control.  In  W.A.  Hershberger  (Ed.),  Volitional  action.  Amsterdam: 
Elsevier/North-Holland,  253-297.  (+) 

7.  Bullock,  D.  and  Grossberg.  S.  (1990).  FLETE:  An  opponent,  neuromuscular  design  for 
factorization  of  length  and  tension.  In  M.  Caudill  (Ed.),  Proceedings  of  the  international 
joint  conference  on  neural  networks,  January.  II.  209-212,  Hillsdale,  NJ:  Erlbaum 
Associates.  (*+) 

8.  Bullock,  D.  and  Grossberg.  S.  (1990).  Motor  skill  development  and  neural  networks  for 
position  code  invariance  under  speed  and  compliance  rescaling.  In  H.  Block  and  B.  Berten- 
thal  (Eds.),  Sensory-motor  organization  and  development  in  infancy  and  early 
childhood.  Dordrecht:  Kluwer  Academic.  (  +  ) 

9.  Carpenter,  G.A.  (19S9).  Neural  network  models  for  pattern  recognition  and  associative 
memory.  Neural  Networks.  2,  243-257.  (*-1-) 
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10.  Carpenter.  G.A.  and  Grossberg.  S.  (1987).  ART  2:  Self-organization  of  stable  category 
recognition  codes  for  analog  input  patterns.  In  M.  Caudill  and”  C.  Butler  (Eds. R  Proceed¬ 
ings  of  the  IEEE  international  conference  on  neural  networks.  II.  727-736.  (  +  “) 

11.  Carpenter,  G.A.  and  Grossberg,  S.  (1987).  Invariant  pattern  recognition  and  recall  by 
an  attentive  self-organizing  ART  architecture  in  a  nonstationary  world.  In  M.  Caudill  and 
C.  Butler  (Eds.).  Proceedings  of  the  IEEE  international  conference  on  neural  net¬ 
works.  II.  737-746.  (+*) 

12.  Carpenter.  G.A.  and  Grossberg.  S.  (1987).  A  massively  parallel  architecture  for  a  self¬ 
organizing  neural  pattern  recognition  machine.  Computer  V ision.  Graphics,  and  Image 
Processing.  37.  54-115.  (  +  x) 

13.  Carpenter,  G.A.  and  Grossberg.  S.  (1987).  ART  2:  Self-organization  of  stabje  category 
recognition  codes  for  analog  input  patterns.  Applied  Optics.  26.  4919-4930.  (+  ) 

14.  Carpenter,  G.A.  and  Grossberg.  S.  (1988).  The  ART  of  adaptive  pattern  recognition  by  a 
self-organizing  neural  network.  Computer.  21.  77— SS.  (  +  ) 

15.  Carpenter,  G.A.  and  Grossberg.  S.  (1988).  Self-organizing  neural  network  architectures 
for  real-time  adaptive  pattern  recognition.  In  H.  Haken  (Ed.),  Neural  and  synergetic 
computers.  New  York:  Springer-Verlag.  42-74.  (  +  *) 

16.  Carpenter.  G.A.  and  Grossberg.  S.  (1989).  Self-organizing  neural  network  architectures  for 
real-time  adaptive  pattern  recognition.  In  S.F.  Zornetzer.  J.L.  Davis,  and  C.  Lau  (Eds^),  An 
introduction  to  neural  and  electronic  networks.  New  \ork:  Academic  Press.  (*+) 

17.  Carpenter,  G.A.  and  Grossberg.  S.  (1990).  Attention,  resonance,  and  transmitter  dy¬ 
namics  in  models  of  self-organizing  cortical  networks  for  recognition  learning.  To  ^appear  in 
Neurocomputers  and  attention.  Manchester:  Manchester  Tniversity  Press.  (  +) 

18.  Carpenter,  G.A.  and  Grossberg,  S.  (1990).  ART  3:  Hierarchical  search  using  chemical 
transmitters  in  self-organizing  pattern  recognition  architectures.  Neural  Networks,  3,  129- 
152.  r+) 

19.  Carpenter,  G.A.  and  Grossberg.  S.  (1990).  ART  3  hierarchical  search:  Chemical  transmit¬ 
ters  in  self-organizing  pattern  recognition  architectures.  In  M.  Caudill  (Ed.),  Proceedings 
of  the  international  joint  conference  on  neural  networks,  January.  II,  30-34,  Hills¬ 
dale,  NJ:  Erlbaum  Associates.  (+*) 

20.  Carpenter,  G.A.,  Grossberg,  S.,  and  Mehanian.  C.  (1989).  Invariant  recognition  of  clut- 
tered  scenes  by  a  self-organizing  ART  architecture:  CORT-X  boundary  segmentation.  Neural 
Networks,  2,  169-181.  (*+) 

21.  Cohen,  M.A.  (1987).  Sustained  oscillations  in  symmetric  cooperative-competitive  neural 
networks.  In  M.  Caudill  and  C.  Butler  (Eds.),  Proceedings  of  the  IEEE  international 
conference  on  neural  networks,  II.  127-130.  (4-) 

22.  Cohen,  M.A.  and  Grossberg.  S.  (1987).  Masking  fields:  A  massively  parallel  architecture 
for  learning,  recognizing,  and  predicting  multiple  groupings  of  patterned  data.  In  M.  Caudill 
and  C.  Butler  (Eds.).  Proceedings  of  the  IEEE  international  conference  on  neural 
networks,  II,  787-794.  (+*) 

23.  Cohen,  M.A.,  Grossberg,  S..  and  Stork,  D.G.  (1987).  Recent  developments  in  a  neural 
model  of  real-time  speech  analysis  and  synthesis.  In  M.  Caudill  and  C.  Butler  (Eds.),  Pro¬ 
ceedings  of  the  IEEE  international  conference  on  neural  networks,  IV,  443-454. 

(+) 

24.  Cohen,  M.A.,  Grossberg,  S.,  and  Stork.  D.G.  (19SS).  Speech  perception,  and  production 
by  a  self-organizing  neural  network.  In  Y.C.  Lee  (Ed.).  Evolution,  learning,  cognition, 
and  advanced  architectures.  Hong  Kong:  World  Scientific  Publishers.  (  +  ) 
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25.  Gaudiano,  P.  and  Grossberg,  S.  (1990).  A  self-regulating  generator  of  sample-and-hoid 
random  training  vectors.  In  M.  Caudill  (Ed.).  Proceedings  of  the  international  joint 
conference  on  neural  networks.  January.  II,  213-216.  Hillsdale,  NJ:  Erlbaum  Associates. 

(  +  ) 

26.  Grossberg,  S.  (19SS).  Nonlinear  neural  networks:  Principles,  mechanisms,  and  architec¬ 
tures.  Neural  Networks ,  1,  17-61.  (  +  ) 

27.  Grossberg,  S.  (1989).  Content-addressable  memory  storage  by  neural  networks:  A  general 
model  and  global  Liapunov  method.  In  E.L.  Schwartz  (Ed.),  Computational  neuro¬ 
science.  Cambridge,  MA:  MIT  Press.  (  +  ) 

28.  Grossberg,  S.  (1989).  Preattentive  visual  perception:  Emergent  segmentation  and  featural 
filling-in.  In  L.  Spillmann  and  J.S.  Werner  (Eds.),  The  neurophysiological  foundations 
of  visual  perception.  New  York:  Academic  Press. 

29.  Grossberg,  S.  (1990).  Self-organizing  neural  architectures  for  motion  perception,  adaptive 
sensory-motor  control,  and  associative  mapping.  In  M.  Caudill  (Ed.),  Proceedings  of  the 
international  joint  conference  on  neural  networks.  January,  II,  26-29,  Hillsdale,  NJ: 
Erlbaum  Associates.  (+x) 

30.  Grossberg,  S.  (1990).  A  neural  network  architecture  for  Pavlovian  conditioning:  Rein¬ 
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A.  ART  2:  Stable  self-organization  of  pattern  recognition  codes  for  analog  input 
patterns  (Articles  A) 

Adaptive  resonance  architectures  are  neural  networks  that  self-organize  stable  pattern 
recognition  codes  in  real-time  in  response  to  arbitrary  sequences  of  input  patterns.  These 
articles  introduce  and  develop  ART  2.  a  class  of  adaptive  resonance  architectures  which 
rapidly  self-organize  pattern  recognition  categories  in  response  to  arbitrary  sequences  of 
either  analog  or  binary  input  patterns.  The  architecture  incorporates  many  new  features 
beyond  those  present  in  the  ART  1  systems,  which  establish  recognition  categories  for  binary 
input  patterns  (Carpenter.  G.  A.  and  Grossberg.  S.,  Computer  Vision,  Graphics,  and  Image 
Processing.  37,  54—1 15,  19S7).  In  order  to  cope  with  arbitrary  sequences  of  analog  input 
patterns.  ART  2  architectures  embody  solutions  to  a  number  of  design  principles,  such  as  the 
stability-plasticity  tradeoff,  the  search-direct  access  tradeoff,  and  the  match-reset  tradeoff.  In 
these  architectures,  top-down  learned  expectation  and  matching  mechanisms  are  critical  in 
self-stabilizing  the  code  learning  process.  A  parallel  search  scheme  updates  itself  adaptively 
as  the  learning  process  unfolds,  and  realizes  a  form  of  real-time  hypothesis  discovery,  testing, 
learning,  and  recognition.  After  learning  self-stabilizes,  the  search  process  is  automatically 
disengaged.  Thereafter  input  patterns  directly  access  their  recognition  codes  without  any 
search.  Thus  recognition  time  for  familiar  inputs  does  not  increase  with  the  complexity  of 
the  learned  code.  A  novel  input  pattern  can  directly  access  a  category  if  it  shares  invariant 
properties  with  the  set  of  familiar  exemplars  of  that  category.  A  parameter  called  the 
attentiencd  vigilance  parameter  determines  how  fine  the  categories  will  be.  If  vigilance 
increases  (decreases)  due  to  environmental  feedback,  then  the  system  automat. cally  searches 
for  and  learns  finer  (coarser)  recognition  categories.  Gain  control  parameters  enable  the 
architecture  to  suppress  noise  up  to  a  prescribed  level.  The  architecture’s  global  design 
enables  it  to  learn  effectively  despite  the  high  degree  of  nonlinearity  of  such  mechanisms. 

B.  Invariant  pattern  recognition:  ART  modules  in  network  hierarchies  (Articles 

B) 

This  work  illustrates  how  ART  modules  may  be  embedded  in  neural  network  hierarchies 

perform  tasks  such  as  invariant  pattern  recognition  and  associative  learning. 

Article  6  describes  a  neural  network  architecture  which  can 

(1)  stably  self-organize  an  invariant  pattern  recognition  code  in  response  to  a  sequence  of 
analog  or  binary  input  patterns; 

(2)  be  attentionally  primed  to  ignore  all  but  a  designated  category  of  input  patterns; 

(3)  automatically  shift  its  prime  as  it  satisfies  internal  criteria  in  response  to  the  occurrence 
of  a  previously  primed  category  of  input  patterns; 

(4)  learn  o  generate  an  arbitrary  spatiotemporal  output  pattern  in  response  to  any  input 
pattern  exemplar  of  an  activated  recognition  category. 

This  architecture  exploits  properties  of  the  ART  1  and  ART  2  adaptive  resonance  theory 
architectures,  and  uses  a  simple  preprocessing  system  based  upon  the  Boundary  Contour 
System  (BCS)  for  boundary  segmentation  (Grossberg.  S.  and  Mingolla.  E.,  Perception  and 
Psychophysics.  38.  141-171.  19S5).  The  system  is  used  to  carry  out  self-organizing,  invariant 
pattern  recognition  of  noisy,  multisensor  laser  radar  images. 

Article  4  describes  recent  progress  on  this  research  project.  In  particular  the  prepro¬ 
cessing  stages  have  been  developed  to  improve  recognition  of  images  with  high  noise  levels. 
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while  retaining  the  rapid  performance  of  a  feedforward  preprocessor.  The  new  circuit  for 
boundary  segmentation,  called  the  CORT-X  filter,  detects,  regularizes,  and  completes  sharp 
(even  one-pixel  wide)  image  boundaries  in  up  to  oQVt  noise,  while  simultaneously  suppressing 
the  noise.  The  CORT-X  filter  achieves  this  competence  by  using  nonlinear  interactions  be¬ 
tween  multiple  spatial  scales  to  resolve  a  design  trade-off  that  exists  between  the  properties 
of  boundary  localization,  boundary  completion,  and  noise  suppression.  The  processing  levels 
of  the  CORT-X  filter  are  analogous  to  those  of  the  Boundary  Contour  System,  but  contain 
only  feedforward  operations  that  are  easier  to  implement  in  hardware.  The  network  nodes 
in  these  levels  are  analogous  to  cortical  simple  cells,  complex  cells,  hypercomplex  cells,  and 
unoriented  and  oriented  cooperative  cells. 

Articles  3.  6,  and  7  outline  how  ART  recognition  modules  can  be  embedded  in  neural 
network  architectures  that  include  recall  and  reinforcement  capabilities.  The  articles  indi¬ 
cate  how  well-developed  individual  network  modules  can  be  linked  to  perform  higher-order 
functions,  including  motor  output. 

C.  Comparative  analysis  and  development  of  neural  network  systems  (Articles 
C) 

These  articles  share  a  goal  of  analysing  various  neural  network  models  by  identifying 
common  themes.  Identification  of  a  core  module  facilitates  comparative  analysis  of  different 
models'  capabilities. 

Review  article  8  outlines  some  fundamental  neural  network  modules  for  associative  mem¬ 
ory.  pattern  recognition,  and  category  learning.  Included  are  discussions  of  the  McCulloch- 
Pitts  neuron,  perceptrons,  adaline  and  madaline,  back  propagation,  the  learning  matrix, 
linear  associative  memory,  embedding  fields,  instars  and  outstars,  the  avalanche,  shunting 
competitive  networks,  competitive  learning,  computational  mapping  by  instar/outstar  fami¬ 
lies.  adaptive  resonance  theory,  the  cognitron  and  neocognitron,  and  simulated  annealing. 
Adaptive  filter  formalism  provides  a  unified  notation.  Activation  laws  include  additive  and 
shunting  equations.  Learning  laws  include  back-coupled  error  correction.  Hebbian  learn¬ 
ing.  and  gated  instar  and  outstar  equations.  Also  included  are  discussions  of  real-time  and 
off-line  modeling,  stable  and  unstable  coding,  supervised  and  unsupervised  learning,  and 
self-organization. 

Article  9  describes  models  of  associative  pattern  learning,  adaptive  pattern  recognition, 
and  parallel  decision-making  by  neural  networks.  It  is  shown  that  a  small  set  of  real-time  non¬ 
linear  neural  equations  within  a  larger  set  of  specialized  neural  circuits  can  be  used  to  study 
a  wide  variety  of  such  problems.  Models  of  energy  minimization,  cooperative-competitive 
decision  making,  competitive  learning,  adaptive  resonance,  interactive  activation,  and  back 
propagation  are  discussed  and  compared. a 

Article  10  shows  how  applications  of  techniques  from  physics,  such  as  energy  minimiza¬ 
tion.  that  have  been  used  in  neural  network  analysis  are  special  cases  of  more  general  results 
in  the  neural  network  literature. 

D.  ART  3:  Hierarchical  search  using  chemical  transmitters  in  self-organizing 
pattern  recognition  architectures  (Articles  D) 

Articles  11  and  12  introduce  and  begin  development  of  a  new  research  project.  The 
ART  3  model  implements  parallel  search  of  compressed  or  distributed  pattern  recognition 
codes  in  a  neural  network  hierarchy.  The  ART  3  search  process  functions  well  with  either  fast 
learning  or  slow  learning,  and  can  robustly  cope  with  asynchronous  processing  of  arbitrary 
sequences  of  input  patterns  in  real-time.  The  search  process  emerges  when  computational 
properties  of  the  chemical  synapse,  such  as  transmitter  accumulation,  release,  inactivation, 
and  modulation,  are  embedded  within  an  ART  2  architecture.  Formal  analogs  of  ions  such  as 
Na+  and  Ca2+  control  nonlinear  feedback  interactions  that  enable  presynaptic  transmitter 
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dynamics  to  model  the  postsynaptic  short  term  memory  representation  of  a  pattern  recogni¬ 
tion  code.  Reinforcement  feedback  can  modulate  the  search  process  by  altering  the  ART  3 
vigilance  parameter  or  directly  engaging  the  search  mechanism.  The  search  process  is  a  form 
of  hypothesis  testing  capable  of  discovering  appropriate  representations  of  a  nonstationary 
input  environment. 

This  work  indicates  solutions  to  computational  bottlenecks  which  can  occur  when  non¬ 
linear  adaptive  feedback  modules,  such  as  ART,  are  embedded  in  neural  network  hierarchies. 
In  particular,  computation  is  required  to  be  robust  and  asynchronous  but  also  capable  of 
rapid  reset. 
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3.  SUMMARY  OF  NEURAL  NETWORK  RESEARCH 

Stephen  Grossberg 

(A)  Some  Earlier  Research  Accomplishments 

1.  A  Self-Organizing  Neural  Network  Architecture  for  Invariant  Pattern  Recog¬ 
nition  and  Nonstationary  Hypothesis  Testing 

Gail  Carpenter  and  Stephen  Grossberg  have  analysed  how  arbitrary  time  series  of  ana¬ 
log  or  digital  input  patterns  can  be  automatically  categorized  and  recognized  by  a  real-time 
Adaptive  Resonance  Theory  (ART)  architecture,  called  ART  2.  Such  an  architecture  has 
been  shown  to  possess  a  unique  set  of  computational  properties  relath'e  to  other  adap¬ 
tive  pattern  recognition  techniques.  See  Table  1  for  a  summary.  In  particular.  ART  2  is 
an  autonomous  hypothesis  testing  and  pattern  recognition  machine  that  is  capable  of  self¬ 
organizing.  self-stabilizing,  and  self-scaling  its  learned  recognition  code.  It  is  capable  of 
directly  accessing  these  codes  after  learning  occurs.  Before  that  happens,  an  efficient  self- 
adjusting  parallel  memory  search  helps  to  discover  a  globally  self-consistent  recognition  code 
during  the  learning  phase. 

Subsequent  to  the  initial  phase  of  this  work,  three  variations  of  the  ART  2  architecture 
were  developed,  all  of  them  exhibiting  the  properties  outlined  in  Table  1. 

With  these  architectures  in  hand,  a  system  was  designed  that  is  capable  of  invariant 
visual  pattern  recognition.  As  in  many  applications  of  ART  2,  a  specialized  preprocessor 
was  designed  to  transform  image  data  before  the  ART  architecture  received  it.  In  this 
application,  Carpenter  and  Grossberg  first  worked  with  Paul  Kolodzy  and  Steve  Rak  of  MIT 
Lincoln  Labs.  Courosh  Mehanian  has  since  joined  Carpenter  and  Grossberg  to  develop  a 
more  advanced  system. 

This  application  uses  a  laser  range  detector  to  compute  image  distance  and  a  laser  doppler 
detector  to  compute  image  velocity.  Intersecting  the  outputs  of  these  detectors  automatically 
separated  image  figure  from  ground. 

The  separated  figure  was  then  passed  through  a  simplified  version  of  a  neural  network 
that  was  developed  by  Grossberg  and  his  colleague  Ennio  Mingolla.  The  Grossberg-Mingolla 
Boundary  Contour  System  (BCS)  detects,  sharpens,  regularizes,  and  completes  image  bound¬ 
aries,  while  suppressing  image  noise. 

These  boundary  representations  were  then  passed  through  a  Fourier- Mellin  filter  to  gen¬ 
erate  spectra  that  are  invariant  under  2-dimensional  image  rotation,  dilation,  and  shift. 

These  invariant  spectra  of  boundary-completed  laser  radar  images  were  the  inputs  to 
the  ART  2  system.  The  ability  of  ART  2  to  classify  any  set  of  analog  imagery  made  this 
application  possible. 

Present  work  is  further  refining  the  ART  2  architecture,  notably  towards  building  a 
many-level  hierarchical  ART  system  capable  of  simultaneously  encoding  an  input  pattern  s 
most  salient  parts,  and  towards  developing  the  system  to  cope  with  high  levels  of  image  noise 
(20%-50%). 

2.  A  Neural  Network  Architecture  for  Self-Organization  of  Speech  Perception 
and  Production  Codes 

This  project  was  undertaken  through  collaboration  of  Michael  Cohen.  Stephen  Gross¬ 
berg.  and  David  Stork.  It  has  led  to  the  articulation  of  a  new  neural  network  architecture,  the 
significance  of  which  is  now  summarized.  Considerations  of  the  real-time  self-organization 
of  neural  networks  for  speech  recognition  and  production  led  to  a  new  understanding  of 
several  key  designs  for  such  networks  which  clarify  how  variable-rate  speech  sounds  of  dif¬ 
ferent  speakers  in  noise  may  be  coherently  completed  and  recognized  in  real-time.  Among 
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TABLE  1 


ART  Architecture 


Real-time  (on-line)  learning 

Nonstationary  world 

Self-organizing  (unsupervised) 

Memory  self-stabilizes  in  response 
to  arbitrarily  many  inputs 

Effective  use  of  full  memory  capacity 

Maintain  plasticity  in  an  unexpected 
world 

Learn  internal  top-down  expectations 

Active  attentional  focus  regulates 
learning 

Slow  or  fast  learning 

Learn  in  approximate-match  phase 

Use  self- regulating  hypothesis  testing 
to  globally  reorganize  the  energy 
landscape 

Fast  adaptive  search  for  best  match 

Rapid  direct  access  to  codes  of 
familiar  events 

Variable  error  criterion  (vigilance 
parameter)  sets  coarseness  of  recognition 
code  in  response  to  environmental 
feedback 

All  properties  scale  to  arbitrarily  large 
system  capacities 


Alternative  Learning  Properties 

Lab-time  (off-line)  learning 
Stationary  world 

Teacher  supplies  correct  answer  (supervised) 

Capacity  catastrophe  in  response 
to  arbitrarily  many  inputs 

Can  only  use  partial  memory  capacity 

Externally  shut  off  plasticity 
to  prevent  capacity  catastrophe 

Externally  impose  costs 

Passive  learning 

Slow  learning  or  oscillation  catastrophe 

Learn  in  mismatch  phase 

Use  noise  to  perturb  system  out  of 
local  minima  in  a  fixed  energy 
landscape 

Search  tree 

Recognition  time  increases  with 
code  complexity 

Fixed  error  criterion  in  response 
to  environmental  feedback 

Key  properties  deteriorate  as  system 
capacity  is  increased 
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the  novel  circuits  that  are  being  further  developed  arc:  the  characterization  of  networks 
of  invariant  auditory  feature  detectors  that  can  process  co-articulated  consonant  and  vowel 
sounds  in  a  speaker-independent  fashion:  the  analysis  of  networks  whereby  a  partially  com¬ 
pressed  auditory  code  is  associatively  mapped  into  articulatory  commands  in  a  manner  that 
permits  imitation  and  code  formation  of  novel  sounds:  the  design  of  top-down  articulatory  - 
to-auditory  priming  signals  that  self-stabilize  the  learning  of  this  imitative  map.  and  provide 
understanding  of  how  speech  is  encoded  by  how  it  would  have  been  produced  (e.g..  mo¬ 
tor  theory);  the  further  compression  of  these  motorically  modified  auditory  codes  into  item 
codes  (e.g..  phonemic  segments):  the  design  of  a  working  memory  for  sequences  of  item 
codes,  which  is  able  to  automatically  adjust  itself  to  cope  with  variable  speech  rates  and 
to  store  its  items  in  real-time  in  a  way  that  preserves  the  stability  of  subsequent  learning 
processes;  the  grouping  of  item  subsequences  from  working  memory  into  compressed  codes, 
or  chunks,  by  a  masking  field  level  whose  top-down  signals  contextually  complete  and  or¬ 
ganize  the  items  in  working  memory  into  coherent  groupings,  as  in  phonemic  restoration 
and  word  superiority  effects.  The  emergent  resonant  wave  that  defines  these  bottom-up  and 
top-down  influenced  working  memory  groupings  through  time  defines  a  speech  code  with 
speaker-  and  rate-independent  invariant  properties.  This  total  architecture  sheds  new  light 
on  key  speech  issues  such  as  coarticulation,  analysis-by-synthesis,  motor  theory,  categorical 
perception,  invariant  speech  perception,  word  superiority,  and  phonemic  restoration. 

3.  Control  of  Predictive  Performance  by  Self-Organizing  Recognition-Reinforce¬ 
ment  Networks 

A  key  problem  in  the  neural  network  literature  concerns  how  sensory  and  cognitive  recog¬ 
nition  codes  interact  with  reinforcement  mechanisms  to  modulate  the  learning  of  recognition 
codes.  This  interaction  reorganizes  recognition  codes  based  upon  the  success  or  failure  of  ac¬ 
tions  based  upon  the  recognition  event.  Reinforcement  calibrates  whether  the  action  has  or 
has  not  satisfied  internal  system  criteria.  Reinforcement  mechanisms  modify  the  formation 
of  recognition  codes  and  shift  attention  to  focus  upon  those  codes  whose  activation  has,  in 
the  past,  generated  actions  that  have  satisfied  internal  system  criteria. 

A  number  of  other  learning  models,  such  as  back-propagation,  have  lumped  together  the 
processes  of  recognition  and  reinforcement  into  a  single  mechanism.  Such  a  step  prevents 
a  model  from  achieving  autonomy  and  stability  of  learning  in  a  statistically  general  input 
environment,  as  indicated  in  Table  1.  A  central  problem  of  neural  network  theory  is  to 
sharply  distinguish  recognition  and  reinforcement  mechanisms,  and  to  design  their  feedback 
interactions  to  achieve  the  modulatory  properties  just  described. 

An  invaluable  guide  in  this  design  process  is  the  huge  data  base  concerning  classical  and 
instrumental  conditioning  that  has  accumulated  over  the  past  half-century.  As  of  this  writing, 
the  conditioning  theory  being  developed  at  CAS  has  provided  principled  explanations  and 
predictions  of  an  order  of  magnitude  more  data  than  alternative  theories  in  this  area.  Its 
computational  power  is  also  correspondingly  more  advanced.  This  theory  is  realized  by  a 
specialized,  hierarchically  organized  ART  model.  Recent  articles  have  further  developed 
this  theory  through  mathematical  analyses  and  extensive  computer  simulations,  which  have 
enhanced  the  theory's  explanatory  and  predictive  range.  Stephen  Grossberg,  Daniel  Levine, 
and  Nestor  Schmajuk  completed  the  first  phase  of  this  work. 

4.  Planned  Arm  Movements  and  Eye-Arm  Coordination 

Daniel  Bullock  and  Stephen  Grossberg  have  shown  how  to  design  a  real-time  neural 
network,  called  the  VTTE  model  (Vector  Integration  to  Endpoint)  for  flexible,  adaptive  arm 
movement  control,  whereby  multijoint  trajectories  can  be  performed  with  synchronous  and 
invariant  properties  at  variable  speeds  and  quickly  updated  without  experiencing  combina¬ 
torial  explosion  or  programming  rigidity.  These  results  have  clarified  how  to  begin  to  design 
self-organizing  associative  maps  to  control  eye-hand  coordination. 
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5.  Multiple  Scale  Automatic  Encoding  of  Distributed  Data  Patterns 

Michael  Cohen  and  Stephen  Grossberg  have  shown  how  to  design  a  masking  field  neu¬ 
ral  architecture;  viz.,  a  multiple-scale  self-similar  automatically  gain-controlled  cooperative- 
competitive  feedback  network  capable  of  generating  predictive  representations  of  uncertain 
distributed  data,  and  therefore  suggesting  a  solution  to  the  credit  assignment  problem. 

6.  Decision  Making  under  Risk 

Stephen  Grossberg  and  William  Gutowski  have  developed  a  model  of  neural  mechanisms 
of  decision  making  under  risk,  that  explains  paradoxical  phenomena  such  as  preference  re¬ 
versals  and  Gambler's  Fallacy. 

7.  Emergent  Segmentation  of  Noisy  Images 

Stephen  Grossberg  and  Ennio  Mingolla  have  demonstrated  how  the  Boundary  Contour 
System  (BCS)  can  regularize  and  complete  sharp  image  boundaries  from  noisy  image  data 
in  a  self-scaling  manner.  This  demonstration  suggests  a  new  approach  to  penetrating  cam¬ 
ouflage.  The  BCS  has  been  used  to  explain  and  predict  a  large  body  of  psychophysical 
and  neurophysiological  data  about  preattentive  vision,  notably  about  texture  segmentation, 
shape-from-shading,  binocular  fusion  and  rivalry,  and  illusory  contours.  It  has  influenced  the 
work  of  many  experimental  laboratories,  and  an  increasing  number  of  its  predictions  have 
been  supported  by  experimental  tests. 

8.  Multiplexed  Multiscale  Fusion  of  Image  Features 

Stephen  Grossberg  and  Jonathan  Marshall  have  shown  how  to  design  an  adaptive  filter 
for  stereoscopic  image  fusion  that  multiplexes  image  data  about  position,  orientation,  spatial 
frequency,  and  binocular  disparity. 

9.  Perception  in  Variable  Illumination 

Stephen  Grossberg  and  Dejan  Todorovic  have  developed  the  2-D  Feature  Contour  System 
(FCS)  that  was  earlier  proposed  in  the  I-D  case  by  Cohen  and  Grossberg.  They  have  shown 
through  extensive  computer  simulations  how  the  model  can  explain  a  large  number  of  visual 
percepts  concerning  brightness  perception,  and  more  generally  the  mechanisms  whereby  the 
brain  can  automatically  compensate  for  variable  illumination  conditions  and  fill-in  surface 
properties.  Grossberg  has  outlined  the  3-D  theory  whereby  surface  properties  of  brightness, 
color,  and  depth  are  nlled-in.  Ken  Johnson  at  Hughes  Aircraft  Company  has  used  the  model 
to  reconstruct  badly  distorted  infrared  images  of  realistic  scenes. 
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(B)  Some  Recent  Research  Projects 

1.  Continuous  Speech  Perception  and  Production 

Michael  Cohen  and  Stephen  Grossberg  have  been  developing  a  simulated  filter  bank  as 
the  first  stage  of  speech  preprocessing  in  the  neural  model  for  continuous  speech  perception. 
The  filter  will  supply  signals  to  subsequent  processing  stages  that  model  the  eighth  nerve  and 
its  target  neural  interactions.  These  stages  disambiguate  coarticulated  consonant  and  vowel 
sounds  based  upon  such  properties  as  their  coherence,  before  the  disambiguated  signals  are 
compressed  into  acoustic  features. 

Cohen  and  Grossberg  are  also  modelling  fundamental  properties  of  the  speech  code;  for 
example,  how  the  speech  code  can  automatically  adjust  its  time  scale  and  can  complete 
locally  ambiguous  data  by  using  top-down  learned  expectations.  These  results  clarify  how 
to  avoid  limitations  of  time  warping  techniques  in  dynamic  programming.  They  also  clarify 
how  speech  signals  from  subsequent  words  can  disambiguate  speech  signals  from  previous 
words  while  sharpening,  rather  than  obfuscating,  word  boundaries,  and  without  disrupting 
the  conscious  ordering  of  perceived  speech  from  past  to  future  words. 

2.  Adaptive  Pattern  Recognition 

Gail  Carpenter  and  Stephen  Grossberg  have  been  further  developing  Adaptive  Resonance 
Theory  (ART)  architectures  for  self-organization  of  distributed  hypothesis  testing  and  pat¬ 
tern  recognition  in  response  to  nonstationarv  streams  of  input  patterns.  They  have  simulated 
a  new  mechanism,  called  ART  3.  for  parallel  search  and  hypothesis  testing  within  a  network 
hierarchy.  This  mechanism  is  capable  of  appropriately  adjusting  the  activities  and  adaptive 
weights  of  distributed  network  codes  in  response  to  disconfirmation  of  a  hypothesis  about  the 
input  pattern.  The  search  mechanism  uses  neural  mechanisms  of  chemical  transmission  and 
nonlinear  signal  transduction  that  were  not  previously  realized  to  be  capable  of  supporting 
a  parallel  hypothesis  testing  cycle. 

3.  Reinforcement  Learning  and  Adaptive  Timing 

Stephen  Grossberg  and  John  Merrill  have  been  developing  neural  models  capable  of  au¬ 
tomatically  learning  to  time  goal-oriented  actions  within  a  reinforcement  learning  paradigm. 
Such  a  competence  is  necessary  to  enable  animals,  humans,  or  freely-moving  robots  to  gener¬ 
ate  actions  that  are  appropriately  delayed  in  order  to  successfully  obtain  goal  objects  under 
real-time  conditions.  The  results  promise  to  shed  new  light  upon  how  the  hippocampus, 
hypothalamus,  septum,  and  cortex  are  designed  and  interact.  They  should  also  prove  useful 
in  the  design  of  freely  moving  adaptive  robots. 

4.  Visual  Motion  Segmentation 

Stephen  Grossberg,  Ennio  Mingolla,  and  Michael  Rudd  are  developing  a  neural  architec¬ 
ture,  called  a  Motion  Boundary  Contour  System,  for  visual  motion  segmentation.  Such  an 
architecture  clarifies  why  parallel  circuits  for  static  segmentation  and  motion  segmentation 
are  needed,  by  computationally  characterizing  novel  properties  of  the  motion  segmentation 
circuit  that  cannot  be  computed  by  a  static  segmentation  circuit.  The  architecture  will  be 
useful  for  generating  coherent  representations  of  rapidly  moving  objects  under  noisy  viewing 
conditions.  It  has  bee  used  to  explain  a  large  body  of  classical  and  recent  data  about  ap¬ 
parent  motion,  motion  capture,  induced  motion,  and  motion  aftereffects  that  other  motion 
models  have  been  unable  to  explain. 

5.  Adaptive  Robotics 

Daniel  Bullock  and  Stephen  Grossberg  are  developing  a  model,  called  the  FLETE  model 
(Factorization  of  LEngth  and  TEnsion)  to  explain  how  neural  commands  can  generate  ac¬ 
curate  goal-oriented  arm  movements  despite  perturbations  by  variable  forces  and  the  inertias 
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generated  by  variable  speeds.  The  model  explains  how  accurate  arm  positions  can  be  com¬ 
manded  at  a  wide  range  of  arm  tensions.  In  so  doing,  it  suggests  a  computational  explanation 
of  spino-muscular  circuit  elements,  such  as  the  size  principle  and  Renshaw  cells,  that  have 
been  mysterious  for  a  long  time.  The  FLETE  model  should  also  be  useful  for  the  control  of 
artificial  robot  arms. 

6.  Self-Organization  of  Motor  Controllers 

Stephen  Grossberg  and  Paolo  Gaudiano  have  been  developing  a  neural  model  for  a 
self-organizing  neural  controller  to  generate  synchronous  trajectories  of  multiple  limb  com¬ 
ponents.  This  work  builds  upon  earlier  work  of  Bullock  and  Grossberg  which  introduced  and 
analysed  the  VITE  model  (Vector  Integration  To  Endpoint)  for  generating  synchronous  limb 
trajectories,  notably  for  arm  movements  and  speech  articulator  movements.  The  present  re¬ 
search  is  studying  how  a  VITE-type  model  can  determine  its  own  best  parameters  to  generate 
optimal  control  of  limb  trajectories.  The  results  have  begun  to  show  how  the  system  can 
spontaneously  generate  actions  that  serve  as  a  basis  for  testing  its  own  parameters  and 
adjusting  them  through  learning  until  they  are  optimized. 
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4.  NEURAL  NETWORK  MODELS  OF  MOTION  PERCEPTION 

Ennio  Mingolla 

In  collaboration  with  Stephen  Grossberg.  Ennio  Mingolla  has  been  developing  a  neural 
architecture  for  segmentation  and  grouping  of  moving  image  contours.  The  model  clarifies 
how  preprocessing  of  motion  signals  by  a  Motion  Oriented  Contrast  (MOC)  Filter  is  joined 
to  long-range  motion  mechanisms  in  a  motion  Cooperative-Competitive  Loop  to  control 
phenomena  such  as  induced  motion,  motion  capture,  and  motion  after-effects.  The  total 
model  system  is  a  motion  Boundary  Contour  System  (BCS)  that  is  computed  in  parallel 
with  the  static  BCS  of  Grossberg  and  Mingolla  before  both  systems  cooperate  to  generate 
a  boundary  representation  for  3-D  visual  form  perception.  The  model  clarifies  how  the 
BCS  used  in  static  segmentation  problems  can  be  modified  for  use  in  motion  segmentation 
problems,  notably  for  analysing  how  ambiguous  local  movements  (the  aperture  problem) 
on  a  complex  moving  shape  are  suppressed  and  actively  reorganized  into  a  coherent  global 
motion  signal.  Unlike  many  previous  approaches,  this  work  analyses  how  a  coherent  motion 
signal  is  imparted  to  all  regions  of  a  moving  figure  (‘’motion  capture  ),  not  only  regions  at 
which  unambiguous  motion  signals  exist. 

Mingolla  has  also  been  collaborating  with  Bryant  ^ork  of  the  Boston  LTniversity  Com¬ 
puter  Science  Department  on  the  development  of  a  Connection  Machine  implementation  of 
the  BCS.  Besides  the  intrinsic  importance  of  this  work  within  the  field  of  parallel  compu¬ 
tation,  the  implementation  promises  to  be  a  key  tool  in  the  development  of  more  detailed 
specification  of  BCS  architectures  for  a  number  of  important  visual  phenomena,  including 
multiple-scale  effects  in  textural  segmentation  and  grouping,  binocular  fusion  and  rivalry, 
and  for  the  use  of  the  BCS  on  natural  images. 

Mingolla  has  continued  to  consult  regularly  with  Michael  Rudd  and  with  James  Todd 
on  their  investigation  of  motion  perception  and  surface  perception.  He  has  given  several 
colloquium  presentations  and  three  invited  talks  at  international  meetings  on  work  sponsored 
by  the  present  grant. 
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5.  RELATED  ACTIVITIES  AT  THE 
CENTER  FOR  ADAPTIVE  SYSTEMS 


1.  Patents 

The  vision  patent  for  the  Boundary  Contour  System  of  Stephen  Grossberg  and  Ennio 
Mingolla  was  approved.  It  was  written  up  in  a  featured  article  in  the  New  York  Times  patent 
section.  A  pattern  recognition  patent  for  ART  2  of  Gail  Carpenter  and  Stephen  Grossberg 
was  also  approved,  as  was  a  robotics  patent  of  Stephen  Grossberg  and  Michael  Kuperstein. 

2.  International  Neural  Network  Society  and  IEEE  International  Conference  on 
Neural  Networks 

Stephen  Grossberg  founded  and  served  as  first  president  of  INN'S.  During  his  presidency, 
members  joined  at  the  rate  of  over  200  per  month,  reaching  a  total  of  3,S00  when  he  left 
office.  Over  600  speakers  and  1700  attendees  were  present  at  the  INNS  first  annual  ny  jting 
held  in  Boston  in  September,  1988. 

Grossberg  was  also  general  chairman  of  the  IEEE  First  International  Conference  on 
Neural  Networks  (ICNN)  that  was  held  in  June.  1987.  He  and  Center  colleagues  played 
a  major  role  in  setting  up  and  carrying  out  the  procedures  for  this  large  and  successful 
meeting.  In  1989,  the  ICNN  and  INNS  meeting  fused  into  the  IEEE/INNS  International 
Joint  Conferences  on  Neural  Networks  (IJCNN).  Thus  the  Center  played  a  central  role  in 
setting  up  this  major  intellectual  resource. 

3.  Neural  Networks  Journal 

Stephen  Grossberg  founded  and  is  co-editor-in-chief  of  the  INNS  journal  Neural  Net¬ 
works,  which  is  published  by  Pergamon  Press.  The  journal  went  from  4  issues  in  1988  to  6 
issues  in  1989.  It  is  widely  respected  as  the  best  journal  in  the  field. 

4.  Graduate  Program  in  Cognitive  and  Neural  Systems 

This  Ph.D.  and  M.A.  granting  program  admitted  its  first  students  in  September,  1988. 
It  developed  a  systematic  graduate  curriculum  of  eight  new  courses  that  cover  all  aspects 
of  neural  network  research.  Students  also  take  a  concentration  of  eight  carefully  selected 
courses  in  a  core  department,  such  as  computer  science  or  mathematics.  Six  professors  were 
hired  into  the  program  in  1988-89. 

The  1988  and  1989  graduate  students  total  40.  The  quality  of  these  student  classes  is 
very  high.  A  number  of  the  students  have  already  begun  to  do  neural  network  research  in 
collaboration  with  a  faculty  advisor. 

5.  Endowed  Wang  Chair 

An  Wang,  founder  of  Wang  Laboratories,  gave  an  endowed  chair  to  Boston  L  niversity, 
which  has  been  awarded  to  Stephen  Grossberg. 

6.  Use  of  Center  Work 

Many  University  researchers,  companies,  and  government  labs  have  been  applying  or 
further  developing  neural  architectures  invented  at  the  Center  for  Adaptive  Systems,  notably 
our  architectures  for  vision,  pattern  recognition,  hypothesis  testing,  CAM,  and  robotics. 
Also,  a  growing  number  of  laboratories  are  successfully  testing  biological  predictions  made 
in  these  models.  These  groups  include  MIT  Lincoln  Laboratory.  Wright-Patterson  Air  Force 
Base.  JPL.  Hughes,  BP,  Booz-Allen  k  Hamilton.  Martin-Marietta.  Raytheon.  Boeing.  TRW, 
SAIC,  Texas  Instruments,  HNC,  NRL,  U.S.  Army  Missile  Command,  and  Joint  Institute  for 
Laboratory  Astrophysics. 
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7.  URI  Vision  Meeting 

A  successful  URI  vision  meeting  was  held  at  Boston  Lniversity  in  March.  19b8.  Its  15 
invited  speakers  included  both  URI  researchers  and  distinguished  external  researchers.  The 
meeting  program  is  found  in  Appendix  A. 

8.  Research  Colloquium 

Distinguished  scientists  lecture  weekly  as  part  of  a  colloquium  that  is  sponsored  by  the 
Center  for  Adaptive  Systems  and  the  graduate  program  in  Cognitive  and  Neural  Systerns. 
The  colloquium  is  advertised  throughout  the  Boston  area  and  draws  an  audience  of  60-T0 
scientists  and  students  from  a  broad  range  of  Boston-area  institutions  whose  composition 
varies  with  the  topic.  The  colloquium  announcements  are  found  in  Appendix  B  to  illustrate 
the  interdisciplinary  range  of  the  speakers  and  their  intellectual  distinction. 

9.  External  Invited  Lectures 

Faculty  at  the  Center  for  Adaptive  Systems  give  75-100  invited  lectures  each  year  on 
their  research,  both  in  the  United  States  and  abroad. 
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FINAL  REPORT:  AFOSR  URI 

Adam  Reeves 
Northeastern  University 
Department  of  Psychology 
360  Huntington  Avenue 
Boston  MA  02115 


Summary  of  Research 

Research  was  proposed  and  undertaken  in  two  areas  of  color  vision:  surface  appearance 
(with  L.  Arend)  and  opponent  processes  (with  S.  Grossberg).  Work  on  spatial  modelling 
was  also  done  (with  J.  Vang). 

Arend  and  I  have  studied  the  perceived  lightness,  brightness,  and  chroma  of  Mondrian- 
like  surfaces  (Arend  and  Reeves.  JOSA  (A)  1743-1751.  1986).  This  work  was  continued 
in  the  present  grant  period  (Ree  ves  tt  al..  1988:  Arend  tt  al..  1990:  Schirillo  et  a/.,  1990). 
We  have  been  primarily  interested  in  mechanisms  of  constancy  (discounting  the  illuminant). 
Both  the  extent  of  constancy  and  its  causes  are  debated.  \on-Kries  adaptation  is  known 
to  produce  good  color  constancy,  but.  when  adaptation  is  controlled,  in  the  earlier  work 
we  found  only  reasonable  constancy  (up  to  60%  of  the  maximum  possible  in  LCS  space) 
when  subjects  match  surface  appearance  ("make  the  target  patch  look  as  if  it  was  cut  from 
the  same  piece  of  paper  as  the  corresponding  patch  ),  \\hen  they  match  the  color  without 
regard  to  surface  quality  (so-called  "unasserted"  matches),  color  constancy  is  residual  (less 
than  20%:  Reeves  tt  al..  1988).  These  results  were  obtained  when  the  stimuli  were  steady 
or  flashed,  and  when  they  were  presented  side-by-side  or  successively.  The  successive  case 
is  particularly  important  because  it  answers  an  objection  to  our  earlier  claim  that  color 
constancy  is  not  automatic:  namely,  that  the  visual  system  computes  the  mean  illuminant 
over  the  entire  display.  If  that  were  the  case,  color  constancy  would  not  be  expected  in  the 
simultaneous  displays:  but  it  still  would  be  expected  in  the  successive  ones.  In  toto,  these 
results  disconfirm  models  which,  like  Retinex,  automatically  discount  the  illuminant  and  so 
cannot  account  for  the  unasserted  matches. 

We  have  also  studies  lightness  and  brightness  constancy  in  displays  involving  depth.  The 
main  result  is  that  lightness  judgments  depend  on  the  perceived  depth  relations,  as  original; 
shown  by  Gilchrist,  but  that  brightness  judgments  are  independent  of  depth  (Reeves,  Schiriiio 
and  Arend,  1987;  Schirillo.  Reeves  and  Arend.  1990).  This  is  true  with  moderate,  large,  and 
very  large  (900:1)  contrast  ratios,  and  with  two  different  psychophysical  methods  (multiple 
choice  and  adjustment)  with  importantly  different  comparison  displays. 

The  second  area  of  research  concerns  modelling  retinal  mechanisms  of  adaptation  and 
opponency.  The  data  of  concern  are  the  Pi’s  earlier  measurements  of  the  effects  of  steady 
and  flickered  adaptations  on  the  sensitivity  of  the  V/B  and  R/G  opponent  pathways  (Reeves, 

1987) .  While  steady  adaptation  can  produce  a  long-lasting  transient  desensitization  (termed 
‘'transient  tritanopia’1  in  the  V/B  case  by  John  Mollon)  just  after  the  field  is  turned  off,  this 
effect  does  not  occur  if  the  adapting  field  is  flickered  at  1  Hz  during  adaptation.  This  para¬ 
doxical  result  cannot  be  predicted  from  the  slow  time  course  implied  by  the  standard  Pugh 
and  Mollon  model  of  transient  tritanopia.  An  initial  two-stage  version  of  Grossberg  s  gated 
dipole  model  shows  suitable  effects  of  flicker  rate  (and  of  duty  cycle),  and  provides  appro¬ 
priate  rebounds  (to  create  transient  desensitization)  at  a  slower  second-level  gate  (Reeves, 

1988) .  This  work,  with  Michael  Rudd  and  Stephen  Grossberg.  is  still  ongoing. 

The  third  area  of  work,  not  originally  proposed,  grew  out  of  an  attempt  to  analyze  the 
spatial  frequency  responses  of  the  ERG  and  YEP  for  linear  and  non-linear  components. 
Rather  surprisingly,  we  found  that  a  single  linear  filter,  with  a  fixed  shape,  could  account 
for  the  great  majority  of  pattern  ERG  data  published  in  the  last  decade  (5ang.  Reeves,  and 
Bearse,  1990),  whether  stimuli  were  gratings,  checks,  or  stripes.  Deviations  from  linearity 
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were  non-systematic.  Contrast,  presentation  frequency,  and  luminance  level  had  no  clear- 
cut  effects  across  studies,  despite  strong  claims  to  the  contrary.  On  the  other  hand.  V  EPs 
recorded  from  the  occiput  do  show  non-linear  responses  (Wu.  Armington.  Yang,  and  Reeves. 
1990).  YVe  have  varied  check  size  and  check  displacement  in  order  to  capture  one  type  of 
non-linearity  in  the  YEP.  which  we  hope  to  model  using  a  thresholded  saturating  response 
function  introduced  by  Grossberg. 

This  research  was  undertaken  with  the  collaboration  of  Lawrence  Arend  of  the  Eye  Re¬ 
search  Institute,  Michael  Rudd  and  Stephen  Grossberg  of  CAS.  and  three  graduate  students 
at  Northeastern.  Jian  Yang,  Marcus  Bearse.  and  Shuang  Wu. 

Abstracts  Supported  by  this  Grant 

Reeves  A..  Schirillo.  J.,  and  Arend.  L.  (19ST).  Depth,  lightness  and  brightness  of  achromatic 
surfaces.  Paper  presented  at  the  annual  meeting  of  OS  A.  October  2.'..  i9S7,  Rochester.  NY. 

Reeves.  A..  Arend.  L..  and  Schirillo.  J.  (1988).  Simultaneous  and  successive  color  constancy. 
Investigative  Ophthalmology  and  Usual  Science ,  29  (ARY  0  Supplement)  162.  19bS. 

Reeves,  A.  ( 19SS).  Fundamental  mechanisms  of  color  vision.  Paper  given  at  the  meeting:  Vi¬ 
sual  form  and  motion  perception:  Psychophysics,  computation  and  neural  networks.  Boston 
University.  March  19S8. 

Reeves.  A.,  Yang.  J.,  and  Bearse.  M.  (19S9).  Spatial  frequency  selectivity  of  the  pattern 
electroretinogram  (PERG).  Investigative  Ophthalmology  and  Visual  Science,  30  (ARVO 
Supplement)  513.  1989. 

YVu.  S.,  Armington.  J.,  Yang.  J..  and  Reeves.  A.  (1990).  Linearity  and  non-linearity  of  visual 
responses  evoked  by  pattern  displacement.  Investigative  Ophthalmology  and  Visual  Science, 
(ARVO  Supplement).  1990. 

Papers  Supported  by  this  Grant 

Arend.  L..  Reeves.  A..  Schirillo.  J..  and  Goldstein.  R.  (1990).  Simultaneous  color  constancy; 
Patterns  with  diverse  Munsell  values.  L'nder  revision:  JOS  A  f.A). 

Reeves.  A.  (1987).  Transient  desensitization  of  opponent  channels:  A  review.  Die  Farbe, 
34.  S9-93. 

Schirillo.  J.,  Reeves.  A.,  and  Arend.  L.  (1990).  Perceived  depth  influences  lightness,  not 
brightness,  of  achromatic  surfaces.  L'nder  revision:  JOSA  (A). 

Yang.  J..  Reeves.  A.,  and  Bearse.  M.  (1990).  Spatial  linearity  of  the  pattern  electroretino¬ 
gram.  Submitted  for  publication  to  Y’jsion  Research. 
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MASSACHUSETTS  INSTITUTE  OF  TECHNOLOGY  Cambridge.  Massachusetts  021 59 

To:  Prof.  Stephen  Grossberg 

Director,  Center  for  Adaptive  Systems 

From:  Whitman  Richards 

Date:  16  February  1990 

RE:  AFOSR  URI  Final 

Discovering  Visual  and  Acoustic  Patterns 


1.  Whitman  Richards,  MIT 

Department  of  Brain  and  Cognitive  Sciences 

The  aim  of  our  research  has  been  to  understand  how  biological 
systems  can  make  strong  assertions  about  their  external  work  from  very 
impoverished  sense  data.  Our  focus  has  been  principally  shape 
recognition,  using  representations  based  upon  curvature  and  singularities 
of  curvature.  Our  methods  for  study  include  both  psychophysics  and 
computational  modelling  and  algorithms.  Recently,  the  emphasis  has 
shifted  from  algorithms  that  are  entirely  "bottom  up"  to  those  which  allow 
the  inclusion  of  "top  down"  knowledge.  This  step  is  needed  to  reduce  the 
complexity  of  problems  associated  with  finding  one  "correct"  interpretation 
of  an  image  from  among  the  many  possible. 

Our  major  advance  over  the  past  year  has  been  to  define  formally 
what  is  meant  by  a  "perception".  Surprisingly,  although  perception  has 
been  studied  for  centuries,  there  has  been  no  definition  which  is  precise 
enough  to  capture  this  notion  or  "event"  as  an  explicit  state  in  a 
computer  program.  Our  definition  hinges  around  how  our  beliefs  and 
knowledge  of  the  world  affect  the  interpretation  of  our  sense  data.  Given 
a  set  of  beliefs,  a  lattice  of  possible  belief  states  can  be  created,  where 
the  modes  in  the  lattice  are  labelled  by  the  beliefs  which  remain  valid  or 
invalid.  A  perception  can  then  be  defined  as  a  local  maximum  in  a  partial 
ordering  within  this  lattice. 

Over  the  past  three  years,  the  URI  grant  has  helped  us  in  several 
ways  other  than  just  supporting  our  current  research  activities,  or  by 
allowing  us  to  maintain  a  modern  computer-based  laboratory.  First,  it  has 
fostered  useful  exchanges  with  the  Boston  participants,  especially 
Grossberg  at  B.U.,  Todd  at  Brandeis  and  Daugman  at  Harvard.  Secondly, 
it  has  enabled  us  to  bring  additional  talent  to  Boston  for  short  periods, 
such  as  Jan  Koenderink  from  Utrecht,  Staffan  Truve  from  Sweden,  and 
in  a  reciprocal  arrangement,  helped  make  possible  an  extremely 
productive  visit  for  Jack  Beusman  to  Koenderink’s  lab  in  Utrecht.  And 
thirdly,  the  grant  has  strengthened  the  bridges  between  activities  in  the 
Brain  and  Cognitive  Sciences  and  EE-CS,  namely,  through  discussions 
between  W.  Richards  and  C.  Searle,  and  shared  students. 
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2.  Campbell  Searle,  M.l.T. 

Department  of  Electrical  Engineering  &  Computer  Science 

Our  aim  has  been  to  understand  how  precise  information  about 
formants  can  be  carried  in  the  auditory  nerve.  Using  the  cat  auditory 
nerve  as  our  model  system,  we  have  completed  a  time-domain  analysis 
of  the  firing  rate  of  over  200  fibers  responding  to  synthetic  speech 
signals.  We  observe  distinct  groups  of  fibers  in  the  neurograms  of  the 
firing-rate  responses.  The  intervals  between  peaks  in  the  firing-rates  of 
the  fibers  in  each  group  are  very  similar,  and  reflect  the  period  of  the 
formant  that  dominates  the  group’s  response.  We  conclude  that  the 
cochlear  filters  have  much  shorter  impulse  responses  than  the  formants 
to  which  they  respond.  Analysis  of  the  intervals  between  peaks  in  the 
firing-rates  confirms  that  these  intervals  correspond  directly  to  the  formant 
periods.  The  overall  formant  estimates  are  better  than  those  of  previous 
spectral  analyses  of  the  neural  data,  and  the  details  of  lower-formant 
dynamics  are  precisely  tracked.  The  direct  representation  of  the  formant 
period  in  the  time-domain  is  contrasted  with  the  diffuse  spectral 
representation  of  the  formant,  the  dependence  of  spectral  peaks  on  non¬ 
formant  parameters,  and  the  distortion  of  the  spectrum  by  physiological 
non-linearities.  Thus,  a  time-domain  analysis  of  the  responses  to  complex 
stimuli  can  be  an  important  addition  to  frequency-domain  analysis  for 
neural  data.  Although  our  findings  bear  most  directly  upon  cochlear 
models  and  the  machine  processing  of  speech,  they  also  indirectly 
suggest  extensions  to  the  scale-space  approaches  used  in  the  study  of 
visual  information  processing. 
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List  of  Publications  (1989-present  only) 


Jepson,  A.  and  Richards,  W.  (1 989)  Perception  and  Perceivers.  Presented  May  25  at 
a  meeting  on  Vision  and  Three  Dimensional  Representations,  University  of 
Minnesota,  Minneapolis.  [To  appear  as  University  of  Toronto,  Dept,  of  Computer 
Science,  Technical  Report.] 

Jepson,  A.  and  Richards,  W.  (1990)  What  is  a  Perception?  To  be  submitted  to 
Cognitive  Science  journal. 

Seeker-Walker,  H.  and  Searle,  C.  (1990)  Time  domain  analysis  of  auditing  nerve 
fiber  firing  rates.  Journal  of  the  Acoustical  Society  of  America  (submitted). 

Wilson,  H.R.  and  Richards,  W.  (1989)  Mechanisms  of  Contour  Curvature 

Discrimination.  Journal  of  the  Optical  Society  of  America.  A  6,  106-115. 
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PART  IV 

Brandeis  University  Summary 
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Visual  Perception  of  Smoothly  Curved  Surfaces 

James  T.  Todd 

The  research  performed  on  this  project  was  designed  to 
investigate  the  basic  mechanisms  by  which  human  observers 
perceive  the  structures  of  smoothly  curved  surfaces.  In  natural 
vision,  there  are  several  different  properties  of  optical 
stimulation  from  which  observers  are  able  to  obtain  information 
about  an  object's  3 -dimensional  form.  These  include  variations 
of  image  motion  called  optical  flow,  variations  in  the  size  and 
density  of  image  contours  called  optical  texture,  and  variations 
in  image  intensity  called  shading.  This  project  employed  a 
variety  of  psychophysical  procedures  to  empirically  investigate 
how  the  human  visual  system  combines  these  different  sources  of 
information  in  order  to  achieve  a  perceptual  representation  of 
objects  in  3-dimensionaJ  space.  The  results  of  these  experiments 
have  also  been  compared  with  the  theoretical  capabilities  and 
limitations  of  numerous  formal  algorithms  used  in  machine  vision 
for  computing  an  object's  3-dimensional  structure  from  patterns 
of  optical  input.  Our  results  indicate  that  human  perception  is 
much  less  accurate  or  precise  than  existing  computational  models, 
but  that  it  is  also  less  dependent  on  assumed  constraints  on  the 
structure  of  the  environment. 
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Publications 


Todd,  J.T.,  Reichel,  F.  &  Mingolla,  E.  (1986).  Perception  of  curved 
surfaces  fran  shading  and  occluding  contours.  Bulletin  of  the 
Psvchoncanic  Society.  24,  349. 

Todd.  J.  T.,  &  Akerstron,  R.  A.  (1987).  The  perception  of  three 

dimensional  form  fran  patterns  of  optical  texture.  Journal  of 
Experimental  Psychology:  Human  Perception  and  Performance.  2,  242- 
255. 

Fetters,  L.  E.,  &  Todd,  J.  T.  (1987).  Quantitative  assessment  of  infant 
reaching  movements.  Journal  of  Motor  Behavior.  19,  147-166. 

Todd,  J.T. ,  Akers tran,  R.A. ,  Reichel,  F.D. ,  &  Hayes,  W.  (1988).  Apparent 
rotation  in  3 -dimensional  space:  Effects  of  temporal,  spatial  and 
structural  factors.  Perception  &  Psychophysics .  43,  179-188. 

Todd,  J.T.  (1988) .  Perceived  3D  structure  from  2-frame  apparent  motion. 
Investigative  Opthamology.  %9,  265. 

Akerstrcm,  R.A. ,  &  Todd,  J.T.  (1988) .  Perception  of  stereoscopic 
transparency.  Perception  &  Psychophysics.  44/  421-432. 

Todd,  J.T.  (1988) .  Minimal  conditions  for  the  perception  of  structure  fran 
motion.  Bulletin  of  the  Psvchancntic  Society.  26,  476-488. 

Todd,  J.T.  (1989).  Models  of  static  form  perception.  In  J.I.  Elkind,  S.K. 
Carol,  J.  Hochberg  and  B.M.  Huey  (Eds.),  Human  Performance  Models  for 
Computer-Aided  Engineering  (pp.  75-88).  Washington,  D.C. :  National 
Academy  Press. 

Todd,  J.T. ,  &  Reichel,  F.D.  (1989).  Ordinal  structure  in  the  visual 

perception  and  cognition  of  smoothly  curved  surfaces.  Psychological 
Review,  96,  643-657. 

Reichel,  F.D. ,  &  Todd,  J.T.  (1989).  Perceptual  depth  inversion  of  smoothly 
curved  surfaces.  Investigative  Opthamology.  30,  252. 

Mingolla,  E.,  &  Todd,  J.T.  (1989).  Perception  of  solid  shape  fran  shading. 
In  B.K.P.  Horn  &  M.J.  Brooks  (Eds.),  Shape  from  shading  (pp.  402- 
442) .  Cambridge,  MA:  MIT  Press. 

Braunstein,  M. ,  &  Todd,  J.T.  (1990) .  On  the  perceptual  distinction  between 
artifacts  and  information.  Journal  of  Experimental  Psychology:  Human 
Perception  and  Performance,  in  press. 

Reichel,  F.D. ,  &  Todd,  J.T.  (1990).  Perceived  depth  reversals  of  smoothly 
curved  surfaces  fran  changes  in  image  orientation.  Journal  of 
Experimental  Psychology:  Human  Perception  and  Performance,  in  press. 

Todd,  J.T.,  &  Reichel,  F.D.  (1990).  The  visual  perception  of  smoothly 

curved  surfaces  fran  double  projected  contour  patterns.  Journal  of 
Experimental  Psychology:  Human  Perception  and  Performance,  in  press. 
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PART  V 


Harvard  University  Summary 


Final  Report,  U.R.I. 

Harvard  Subcontract  (J.G.  Daugman) 


I.  Publications  partially  supported  under  AFOSIt  U.R.I.  contract: 

Monograph: 

Daugman,  J.G.  Signal  Processing  by  Neural  Mechanisms.  Toshiba  Endowed  Chair 
Lecture  Series,  Tokyo  Institute  of  Technology,  Department  of  Computer  Science 
(Inaugural  Volume).  249  pages.  Tokyo:  Tokyo  Institute  of  Technology  Press,  1990. 

Articles: 

Daugman,  J.G.  (1990)  An  information-theoretic  view  of  analog  representation  in  striate 
cortex.  In  Computational  Neuroscience  (E.  Schwartz,  editor).  Cambridge:  M.I.T. 
Press. 

Daugman,  J.G.  (1990)  Brain  metaphor  and  brain  theory.  In  Computational  Neuro¬ 
science  (E.  Schwartz,  editor).  Cambridge:  M.I.T.  Press. 

Daugman,  J.G.  (1989)  Entropy  reduction  and  decorrelation  in  visual  coding  by  ori¬ 
ented  neural  receptive  fields.  I.E.E.E.  Transactions  on  Biomedical  Engineering ,  36, 
pp.  107-114. 

Daugman,  J.G.  (1989)  Non-orthogonal  wavelet  representations  in  relaxation  networks: 
image  encoding  and  analysis  with  biological  visual  primitives.  In  N  euro  computing 
(John  Taylor,  editor).  London:  Institute  of  Physics  Press. 

Daugman,  J.G.  (1989)  Networks  for  image  analysis:  motion  and  texture.  I.E.E.E. 
Proc.  I.J.C.N.N.,  89  1-5. 

Daugman,  J.G.  (1988)  Complete  discrete  2-D  Gabor  transforms  by  neural  networks 
for  image  analysis  and  compression.  I.E.E.E.  Transactions  on  Acoustics,  Speech, 
and  Signal  Processing,  36,  pp.  1169-1179. 

Daugman,  J.G.  (1988)  Book  Review:  The  Adaptive  Brain  (S.  Grossberg,  editor).  Math¬ 
ematical  Biosciences,  91,  pp.  113-116. 

Daugman,  J.G.  (1988)  Relaxation  neural  network  for  non-orthogonal  image  transforms. 
I.E.E.E.  Proc.  I.C.N.N.  88-1,  pp.  547-560. 

Daugman,  J.G.  (1988)  Pattern  and  motion  vision  without  Laplacian  zero-crossings. 
Journal  of  the  Optical  Society  of  America  -  A,  5,  pp.  1142-1148. 
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Daugman,  J.G.  (1987)  Image  analysis  and  compact  coding  by  oriented  2-D  Gabor 
primitives.  Proceedings  of  the  International  Society  for  Optical  Engineering,  758, 
pp.  19-30. 

Daugman,  J.G.  and  Kammen,  D.M.  (1987)  Pure  orientation  filtering:  A  scale-invariant 
image  processing  tool  for  perception  research  and  data  compression.  Behavior  Re¬ 
search  Methods,  Instruments,  and  Computers ,  18,  pp.  559-564. 

Kammen,  D.M.,  and  Yuille,  A.L.  (1988)  Spontaneous  symmetry-breaking  energy  func¬ 
tions  and  the  emergence  of  orientation  selective  cortical  cells.  Biological  Cybernetics 
59,  pp.  23-31. 

Ph.D.  Dissertations  supported  by  U.R.I.: 

Kammen,  Daniel  M.  Self-Organization  in  Neural  Networks.  (Supervisor:  J.G.  Daug¬ 
man).  Physics  Department,  Harvard  University.  Ph.D.  awarded  June  1988. 

Lee,  Tai-Sing.  Representation,  Generation,  and  Coordination  of  Computational  Maps 
in  Primate  Striate  Cortex.  (Supervisor:  J.G.  Daugman).  Engineering  Sciences  De¬ 
partment,  Harvard  University.  Ph.D.  expected  June,  1991. 

Zheng,  Liu.  Neural  Encoding  of  Motion  in  Fly  Lobula  Plate.  (Supervisor:  J.G.  Daug¬ 
man;  masters  paper).  Engineering  Sciences  Department,  Harvard  University.  Ph.D. 
expected  June,  1992. 

Noteworthy  prize-winning  undergraduate  Theses  also  supported  by  U.R.I.: 

O’Reilly,  Randall  C.  Properties  of  a  Self-Organizing  Neural  Network:  The  Unsupervised 
Generation  of  Hierarchical  Representations.  (Supervisor:  J.G.  Daugman).  Earned 
B.A.  Summa  Cum  Laude ,  Harvard  University,  1989.  Won  Hoopes  Prize,  as  well  as 
Harvard  Faculty  Prize. 

Seybold,  John  L.C.  Connectionist  Theory  of  Human  Categorization  Based  on  Hop- 
field  Networks.  (Supervisor:  J.G.  Daugman).  Earned  B.A.  Summa  Cum  Laude, 
Harvard  University,  1988.  Won  Harvard  Faculty  Prize  and  Rhodes  Scholarship. 
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II.  Summary  of  Research  Progress: 

(a).  Invertebrate  Neural  Mechanisms  of  Figure  /  Ground  Segregation 
(see  Videotape) 

The  rich  visuo-motor  behavioral  repertoire  of  simple  invertebrates  can  be  elicited 
by  motion  discontinuities  in  the  visual  field.  The  demarcation  of  different  velocity 
vector  fields  in  the  retinal  stimulus  array,  even  in  the  absence  of  any  other  cues  (lu¬ 
minance,  contrast,  density,  chrominance,  disparity),  constitutes  a  sufficient  cue  for 
visual  figure/ground  segregation.  With  physical  motion  through  the  environment 
over  time,  such  discontinuities  in  the  retinal  velocity  vector  field  can  reveal  the  3-D 
spatial  configuration  of  the  environment. 

The  Harvard  group  has  built  a  neurophysiology  laboratory  for  recording  from  iso¬ 
lated  neurons  in  the  lobula  plate  of  the  Sarcophagus  Blowfly,  stimulated  visually 
with  figure/ground  moving  texture  fields.  We  use  conventional  recording  techniques 
but  novel  stimulation  hardware  of  our  own  design,  duplicate  copies  of  which  have 
recently  been  installed  also  in  the  world’s  leading  invertebrate  neural  laboratories 
(Reichardt  in  Tubingen,  and  Horridge  in  Canberra).  The  dedicated  hardware,  which 
is  built  into  Programmable-Logic  Array  chips  clocked  at  30  Megahertz,  permits  the 
generation  of  different  2-D  velocity  vector  fields  in  different  image  regions.  The  very 
feist  30  Megahertz  clock  permits  a  high  frame  rate  of  200  Hz,  which  is  a  prerequisite 
for  generating  smooth  motion  displays.  Both  “figure”  and  “ground”  texture  fields, 
defined  by  specified  boundaries,  can  be  assigned  any  2-D  velocity  vector  field. 

The  fly  visual  system  processes  motion  in  Cartesian  vector  form.  Two  orthogo¬ 
nal  classes  of  motion  neurons,  V(ertical)  and  H(orizontal)  cells,  exist  in  the  lobula 
plate.  Each  one  integrates  motion  cues  across  the  entire  contralateral  retina,  in 
the  form  of  vector  projections  onto  these  two  basis  vectors.  By  generating  moving 
texture  fields  whose  two  Cartesian  velocity  vector  coordinates  can  be  independently 
manipulated,  we  have  been  able  to  study  the  interactions  between  the  vertical  and 
horizontal  motion  vector  coordinates  in  the  neural  matrix.  Specifically,  we  have 
measured  the  firing  rates  of  individual  lobula  plate  neurons  stimulated  by  moving 
texture  fields  whose  velocity  vectors  have  the  form  5 j  =  ( vx,vy ),  S2  =  (vx,0),  and 
5s  =  (0,v„).  All  three  velocity  fields  have  different  speeds  and  different  directions 
of  motion,  but  S\  and  S2  have  the  same  horizontal  component  of  motion,  while  Si 
and  S3  have  the  same  vertical  component  of  motion.  Thus  the  classical  framework 
for  understanding  the  fly  visual  motion  system  predicts  the  same  response  from  an 
H  neuron  for  stimuli  S\  and  S2 ,  and  the  same  response  from  a  V  neuron  for  stimuli 
Si  and  S3,  despite  all  the  differences  among  these  stimuli  in  speeds  and  directions. 
We  have  shown  that  this  is  true  over  small  angles,  but  we  have  been  able  to  demon¬ 
strate  definitive  inhibitory  interactions  between  the  vertical  and  horizontal  motion 
systems  over  larger  angles.  Thus  Cartesian  vector  projection  of  the  independent 
velocity  vector  components  onto  the  H  and  V  neural  sub-systems  is  not  an  adequate 
model,  and  competitive  interactions  must  be  incorporated. 
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(b).  Studies  of  Human  Visual  Motion  Perception  (see  Videotape) 


An  apparent  paradox  exists  in  motion-based  visual  figure/ground  segregation.  The 
measurement  of  motion  in  the  stimulus  array  requires  an  interval  of  time  and  a 
region  of  space,  with  greater  uncertainty  in  the  motion  estimate  resulting  from  the 
narrowing  of  either  of  these.  However,  we  perceive  motion  discontinuity  boundaries 
as  phenomenally  sharp,  even  with  very  small  difference  vectors  between  the  two 
velocity  fields.  (See  videotape).  There  should  be  an  uncertainty  principle  limitation 
here.  How  can  the  visual  system  make  a  precise  measurement  of  2-D  velocity  vector 
fields,  and  simultaneously  assign  a  crisp  boundary  in  space  (and  in  time)  to  the 
discontinuity  between  the  two  velocity  signals? 

We  have  investigated  the  parameters  of  motion-based  figure/ground  segregation 
in  human  observers.  Using  only  velocity  cues,  with  all  other  parameters  of  the 
moving  texture  fields  identical,  we  have  measured  the  magnitudes  of  the  differences 
in  speed  and/or  direction  between  the  two  motion  vectors  necessary  to  produce  a 
figure/ground  percept.  These  tend  to  be  Weberian  (differential  signal  proportional 
to  vector  norm).  However,  there  are  significant  differences  between  the  efficacy  of 
a  speed  differential,  and  the  efficacy  of  a  direction  differential,  in  producing  a  fig¬ 
ure/ground  segregation  percept.  We  have  also  begun  to  map  out  the  necessary  and 
sufficient  vector  differentials  for  driving  a  filling  in  process  based  on  the  differential 
velocity  vector  field,  versus  only  seeing  the  1-D  boundary  contour. 

A  parallel  research  project  now  underway  concerns  the  perception  of  XOR’ed  mov¬ 
ing  texture  fields.  This  is  a  new  form  of  motion  transparency,  in  which  two  2-D 
moving  binary  texture  fields  are  combined  through  the  Exclusive-OR  boolean  oper¬ 
ator  (essentially  multiplicative,  rather  than  additive)  by  our  special  30  Megahertz 
hardware.  The  very  fact  that  two  independent  motion  fields  can  be  seen  at  all  is 
noteworthy,  since  such  stimuli  contain  absolutely  no  Fourier  motion  energy  in  any 
direction.  Nonetheless,  the  dual  motion  percepts  are  utterly  salient.  Thus  this  novel 
motion  stimulus  has  considerable  theoretical  significance;  no  existing  motion  models 
appear  to  be  capable  of  capturing  it.  We  have  begun  to  map  out  the  differential 
vector  parameters  which  are  necessary  and  sufficient  to  drive  this  new  class  of  mo¬ 
tion  transparency  percept. 

Finally,  as  another  challenge  to  existing  motion  models,  we  generated  a  counter¬ 
example  to  the  popular  (“M.I.T.”)  model  of  motion  mechanisms  based  on  the 
movement  of  Laplacian  zero-crossings  within  the  stimulus  array.  (See  “Pattern 
and  Motion  Vision  without  Laplacian  Zero-Crossings,”  JOS  A  1988.)  We  generated 
families  of  moving  textures  which,  at  all  spatial  scales  of  analysis,  have  only  sta¬ 
tionary  Laplacian  zero-crossings.  Convolution  of  these  spatio-temporal  stimuli  with 
V2Ga(x,y)  operators,  of  all  different  blurring  scales  <r,  produces  an  output  all  of 
whose  zero-crossings  are  exactly  stationary.  Nonetheless,  the  motion  of  the  stimulus 
is  clearly  perceived. 
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(c).  Relaxation  Computation  of  Non-Orthogonal  Image  Transforms 


It  is  often  desirable  in  image  processing  to  represent  image  structure  in  terms  of  a  set 
of  coefficients  on  a  family  of  expansion  functions.  For  example,  familiar  approaches 
to  image  coding,  feature  extraction,  image  segmentation,  statistical  and  spectral 
analysis,  and  compression,  involve  such  methods.  It  has  invariably  been  necessary 
that  the  expansion  functions  employed  comprise  an  orthogonal  basis  for  the  image 
space,  because  the  problem  of  obtaining  the  correct  coefficients  on  a  non-orthogonal 
set  of  expansion  functions  is  usually  arduous  if  not  impossible.  Oddly  enough,  image 
coding  in  biological  visual  systems  clearly  involves  non-orthogonal  expansion  func¬ 
tions.  Indeed,  it  would  be  nearly  impossible  to  satisfy  the  rigid  constraints  of  precise 
control  over  center  positions,  weighting  structure,  and  overlap  factors,  that  orthog¬ 
onality  would  require  of  receptive  fields,  given  their  notorious  “scatter”  in  all  of 
these  parameters.  The  receptive  field  profiles  of  visual  neurons  with  linear  response 
properties  typically  have  large  overlaps  and  large  inner  products,  and  are  suggestive 
of  a  conjoint  (spatial  and  spectral)  “2-D  Gabor  representation.”  The  2-D  Gabor 
transform  has  useful  decorrelating  properties  and  provides  a  conjoint  image  descrip¬ 
tion  resembling  a  speech  spectrogram,  in  which  local  2-D  image  regions  are  analyzed 
for  orientation  and  spatial  frequency  content,  but  its  expansion  functions  are  non- 
orthogonal.  We  have  developed  a  general-purpose  three-layered  relaxation  “neural 
network”  that  efficiently  computes  the  correct  coefficients  for  this  and  other,  non- 
orthogonal,  image  transforms.  Examples  of  applications  in  image  analysis  include: 
(1)  image  compression  to  around  0.3  bit/pixel;  and  (2)  textural  image  segmentation 
based  upon  the  statistics  of  the  2-D  Gabor  coefficients  found  by  the  relaxation  net¬ 
work. 

(d).  Properties  of  Self-Similar  2-D  Gabor  Wavelet  Representations 

Building  upon  the  relaxation  network  approach  described  above,  several  different 
schemes  of  image  analysis  and  representation  have  been  explored.  Once  the  re¬ 
strictive  constraint  of  orthogonality  has  been  lifted,  many  new  approaches  become 
possible  which  were  previously  prohibited  by  the  lack  of  efficient  means  for  obtain¬ 
ing  the  coefficients  that  constitute  the  image  representation.  Accordingly,  we  have 
explored  several  different  image  codes  based  on  self-similar  2-D  Gabor  “wavelets," 
which  obey  a  generative  family  of  equations  for  self-similarity  under  dilation,  rota¬ 
tion,  and  translation.  Thus  all  members  of  this  family  can  be  generated  by  dilations 
and  shifts  of  a  single  basic  wavelet.  A  particular  focus  of  work  has  concerned  the 
trade-off  between  the  numbers  of  discrete  orientations  employed  in  the  representa¬ 
tion,  and  the  number  of  positions.  We  have  shown  that  as  long  as  the  number  of 
linearly  independent  degrees-of-freedom  remains  fixed,  good  image  representations 
can  be  obtained  with  many  different  variations  in  the  sampling  rules  for  these  di¬ 
mensions. 

All  wavelet  schemes,  including  the  present  non-orthogonal  one,  are  parameterized 
by  a  geometric  scale  parameter  m  and  position  parameter  n  which  relate  members 
of  the  family  to  each  other.  In  the  classic  one-dimensional  case  extensively  studied 
by  French  mathematicians  Yves  Meyer,  Ingrid  Daubecies,  Alex  Grossmann,  and 
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Stephane  Mallat, 


<^mn{x)  =  2-m^<i/{2-rnx  -n).  (1) 

Generalizing  to  two  dimensions  and  incorporating  discrete  rotations  9  into  the  gen¬ 
erating  function,  together  with  shifts  p,q  and  dilations  m,  the  present  2-D  Gabor 
“wavelet”  set  can  be  generated  from  any  given  member  by: 

*mpqe(x,y)  =  2-m*(x'>y')  (2) 

where 

x'  =  2-m[xcos(0)  +  ysin(0)]  -  p  (3) 

y  =  2~m[-x  sin(0)  +  y  cos(0)j  -  q  (4) 

By  using  the  relaxation  network  to  find  optimal  coefficients  on  this  self-similar  multi¬ 
resolution  wavelet  scheme,  in  which  2-D  Gabor  elementary  functions  serve  as  the 
y),  we  have  been  able  to  explore  many  new  aspects  of  orientation-based, 
multi-scale,  self-similar  image  codes. 

One  recent  application  of  these  principles  is  an  iris-based  biometric  signature  se¬ 
curity  system  for  automatic  visual  personal  identification.  This  very  efficient  algo¬ 
rithm  establishes  the  identity  of  an  individual  in  less  than  a  second  by  examining 
the  detailed  textural  structure  of  his  iris,  registered  by  a  telescopic  videocamera  as 
an  “optical  fingerprint”  without  any  physical  contact.  The  detailed  iris  structure  is 
extracted  at  many  scales  of  analysis  by  projection  into  coefficients  on  a  self-similar 
set  of  quadrature  2-D  Gabor  functions.  A  working  prototype  of  this  machine  has 
been  built  and  demonstrated  to  representatives  of  Federal  security  agencies.  Pre¬ 
liminary  tests  show  that  this  “biometric  signature  security  system”  outperforms  all 
other  automatic  personal  identification  systems  by  about  an  order-of-magnitude. 
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Traditional  Form  and  Motion  Stimuli 
Presented  to  Isolated  Cone  Classes 

Robert  L.  Savoy  ud  Michael  M.  Burns 

.  Rowland  Institute  for  Science 

100  Cambridge  Parkway 
Cambridge.  .V/.4  02142 

Two  ideas  are  explored,  both  of  which  concern  the  organization  of  cone  classes  in  the  subsequent 
connections  of  the  visual  system. 

1:  The  simplest  notion  of  isoluminance  is  problematic.  If  isoluminance  is  determined  by  a 
weighted  sum  of  the  input  from  individual  cone  classes  then  all  the  phenomena  reported  to  occur 
with  isoluminant  stimuli  should  be  present  whatever  the  spectral  composition  of  the  colors  used. 
Thus,  for  example,  the  disappearance  of  the  kinetic  depth  effect  should  occur  equally  well  using  green 
dots  on  a  grey  background  or  green  dots  on  a  red  background.  Our  work  demonstrates  that  this  is  not 
the  case.  Using  both  subjective  and  objective  criteria,  the  quality  of  isoluminance  matches  (and  the 
psychophysical  phenomena  associated  with  isoluminant  stimuli)  vary  dramatically  with  the  spectral 
composition  of  the  colored  stimuli.  The  subjective  measure  is  simply  the  reported  quality  of  match. 
The  objective  measure  is  the  variability  of  isoluminance  settings.  In  particular,  when  the  colors  are  a 
grey  and  green,  the  accuracy  (repeatability)  of  isolumance  matches  is  extremely  good — about  1%  (as 
good  as  making  a  luminance  match  with  a  single  hue).  And  subjects  experience  the  match  as  being 
excellent — in  the  case  of  the  kinetic  depth  effect,  all  sense  of  depth  is  lost  at  isoluminance.  On  the 
other  hand,  when  the  colors  are  a  saturated  red  and  saturated  blue,  the  variability  of  isoluminance 
settings  is  much  higher — about  15%.  The  subjective  phenomena  are  much  weaker  as  well.  Subjects 
continue  to  experience  such  phenomena  as  the  kinetic  depth  effect  at  all  relative  settings  of  the  two 
colors  (in  contrast  to  some  reports).  This  finding  is  robust  across  a  number  of  criteria — the  kinetic 
depth  effect,  stereopsis,  hetereochromatic  brightness  matches,  and  flicker. 

2:  It  is  possible  (using  computer  generated  stimuli)  to  present  a  broad  range  of  traditional  psy¬ 
chophysical  stimuli  so  that  only  a  single  clas3  of  cones  can  be  used  in  the  perception  of  those  stimuli. 
This  is  an  extension  of  the  body  of  work  done  using  the  “cardinal”  directions  of  color  space,  to  use 
the  term  coined  by  Krauskopf,  et  al.  Stimuli  are  designed  so  that  the  spatial  and  temporal  pattern  on 
(for  instance)  two  of  the  three  cone  classes  is  uniform.  The  third  cone  class  is  presented  with  spatial 
and  temporal  patterns  of  arbitrary  complexity.  The  simplest  experiment  is  to  measure  the  contrast 
threshold  for  detection  of  a  spot  by  a  single  cone  class.  The  results  (which  replicate  the  work  of  sev¬ 
eral  other  laboratories)  reveal  the  cardinal  directions  to  be  one  luminance-like  channel  and  two  color 
difference  channels.  More  general  stimuli  are  possible.  For  instance,  one  can  present  a  random  dot 
stereogram  so  that  the  pattern  on  the  left  eye  is  seen  only  by  the  long-wave  cones  while  the  pattern  on 
the  right  eye  is  seen  only  by  the  middle  wave  cones.  Stereopsis  is  still  obtained  (and  can  be  quantita¬ 
tively  assessed,  in  terms  of  the  contrast  needed  to  be  perceived).  More  than  one  intensity  level  can  be 
presented  at  once,  so  that  Mach  Band  patterns,  Simultaneous  Contrast  patterns,  and  even  Mondrian- 
like  patterns  can  be  presented  to  isolated  cone  classes.  The  only  restriction  is  that  the  maximum 
contrast  cannot  be  too  large — about  15%  on  the  long  and  middle  wave  cone  classes  and  about  80% 
on  the  short  wave  cones.  This  limitation  is  imposed  by  the  overlap  of  the  cone  pigment  absorptions. 
These  techniques  for  isolating  the  activities  of  cone  classes  have  already  been  used  in  some  neuro¬ 
physiological  work  and  promise  to  be  a  useful  tool  in  determining  connections  in  the  visual  system. 
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